Repository navigation
Build the FIPS profile in CI, and make it build again - #1009
fredericgermain wants to merge 4 commits into
Conversation
f5cb588 to
a7fd9c2
Compare
|
Still WIP, the last validation run is in progress. I'll come back at it tonight. Not critical for a release IMHO, but nice to have a better support for FIPS in latest release
For discussion: a certified pin means shipping two-year-old crypto source, which contradicts normal patch hygiene. The 2025 modules sit at "Comment Resolution" on NIST's in-process list 10 to 20 months after their version dates, and the CMVP's stated goal of a six-month queue by end of 2025 and none by mid-2026 (OpenSSL Conference, Oct 2025) is not happening. FIPS.md now documents an update stream, main with -DFIPS=1, per the FedRAMP policy. |
a7fd9c2 to
63ab2f1
Compare
|
@fredericgermain let me know once this is ready |
|
Hi @normanmaurer, Let me try to finish that this week. |
63ab2f1 to
114a502
Compare
Motivation: netty#997 made the boringssl-static native-jar step run `patchelf --remove-needed` on Linux, but only the two CI images got the binary. The `build` service of the Arch and openSUSE images has failed since with "patchelf: not found". CI does not run them, so nothing flagged it. Modifications: Add the distro's `patchelf` package to both images. The Debian 7 image is left alone: wheezy packages no patchelf, and its GCC 4.9 cannot build BoringSSL's C++17 anyway, so only its dynamic-only service, the one CI runs, is usable. Result: `docker compose ... run build` works again on Arch and openSUSE.
Motivation: The fips-boringssl-static profile no longer builds from a clean checkout: - Its source tarball on commondatastorage.googleapis.com/chromium-boringssl-fips is no longer public (403). Downstream builds only work off download-maven-plugin's cache. - netty#950 dropped CMAKE_POSITION_INDEPENDENT_CODE. BoringSSL only sets it for the bcm targets, so the aarch64 link fails: "relocation R_AARCH64_ADR_PREL_PG_HI21 ... recompile with -fPIC". It also runs a bare `ninja`, which builds all 564 BoringSSL targets, mostly tests nobody runs, and it hardcodes clang-12. Modifications: - Clone BoringSSL with maven-scm and check out a pinned sha, as the default profile does. The pin is the head of fips-20260721, Google's newest FIPS branch (see FIPS.md). The sha goes into the jar manifest. - -DCMAKE_POSITION_INDEPENDENT_CODE=TRUE, as the default profile passes. - ninja builds crypto, ssl, decrepit and bssl only. - fipsCC / fipsCXX properties, default `clang` / `clang++`. Result: The profile builds from a clean checkout, in minutes. The module it contains holds no certificate yet.
Motivation: Built on a current distro (Debian 13 here) instead of the CentOS release images, the artifact imports glibc-only symbols. It still loads on Alpine under the JVM, which binds lazily, but `ldd`, LD_BIND_NOW and docker/musl-verify fail. Building with an empty musl_compat.c and asking Alpine's loader what is missing gives six symbols: - __isoc23_strtol (APR), __isoc23_strtoul (static libstdc++), __isoc23_strtoull (BoringSSL): glibc 2.38 redirects strtol and friends there under _GNU_SOURCE, and no -D switches it off. - __libc_single_threaded: static libstdc++, glibc 2.32+. - _dl_find_object: libgcc_eh.a from gcc 12, glibc 2.35+. - arc4random: libstdc++'s std::random_device, glibc 2.36+, dragged in by the FIPS profile's newer BoringSSL. Nothing calls it. Modifications: - FIPS link line: -Wl,--gc-sections. The linker drops the unreachable random_device code and the arc4random import with it; the library gets 117 KB smaller and the FIPS integrity check still passes. - Weak fallbacks in musl_compat.c for the other five, which sit in live code. The strto* ones reach the plain symbols through asm labels, since a literal strtol() there would be redirected too and recurse on musl. `__restrict`, not `restrict`: Debian 7's GCC 4.9 is gnu90. - scripts/check_musl_compat.sh asserts the fallbacks are defined, and knows the __isoc99_* scanf names musl does export. Result: Artifacts built on Debian 13 resolve completely on Alpine. No-op on the CentOS release images, whose glibc predates all of these.
114a502 to
a739729
Compare
|
@normanmaurer this is ready for review.
It took a few turns to get here, so here is the path, in case the end result looks arbitrary:
On musl: built on a modern distro the artifact needs a few glibc-only fallbacks in That FIPS build is quite close to something I would consider taking to production. The one thing I am not sure about is whether the compilation differences (clang, FIPS mode, the delocated module) have a bad impact on performance. I plan to run some A/B benchmarks against the normal build, hopefully I find the time. Two things for discussion:
The commits are self-contained if you would rather take the Arch and openSUSE patchelf fix separately. |
| </execution> | ||
| </executions> | ||
| <configuration> | ||
| <url>https://commondatastorage.googleapis.com/chromium-boringssl-fips/boringssl-${boringsslBranch}.tar.xz</url> |
There was a problem hiding this comment.
It's the tar bundle that's actually FIPS validated, isn't it? I'm not sure we can just jump to the latest commit on the boringssl fips branch.
There was a problem hiding this comment.
From https://boringssl.googlesource.com/boringssl/+/refs/heads/main/crypto/fipsmodule/FIPS.md
The last validated module is 2024-08-05 (fips-20240805), certificate (#5244, issued April 2026)
It was updated already to fips-20251031 previously in netty-tcnative. I believe this had more to do to get newer API and avoid compilation problem. So the current version is not fips validated.
FIPS.md recommand to use main directly.
Also to be noted, https://commondatastorage.googleapis.com/chromium-boringssl-fips is not working anymore, we need to fetch git commits directly on the upstream repo.
| && tar -C /opt -xzf go$go_version.linux-$ARCH.tar.gz && rm go$go_version.linux-$ARCH.tar.gz \ | ||
| && ln -s /opt/go/bin/go /usr/local/bin/go && ln -s /opt/go/bin/gofmt /usr/local/bin/gofmt \ | ||
| && ln -s /usr/lib/jvm/java-21-openjdk-$ARCH /usr/lib/jvm/java-21 \ | ||
| && go version && clang --version | head -1 && ninja --version && cmake --version | head -1 |
There was a problem hiding this comment.
I think this runs in a sub-shell that doesn't have pipefail, so the exit code of e.g. clang --version | head -1 will actually be from head -1 instead of clang --version.
I think it's harmless to just drop the | head -1 part.
There was a problem hiding this comment.
Right, the pipe is hiding the exit status of clang --version and cmake --version. Dropped both | head -1.
| # Not pinned to linux/amd64 like the older images, so it can also be built natively on an | ||
| # arm64 host for a quick local run. CI builds it on amd64 runners. | ||
| # | ||
| # Unlike the CentOS 6 image this is NOT a release builder: its artifact has a glibc 2.34 floor. |
There was a problem hiding this comment.
glibc 2.34 floor
In the readme you say 2.41?
There was a problem hiding this comment.
The README was wrong. 2.41 is the image's own glibc, the artifact floor is 2.34.
Fixed.
Well done on catching this.
There was a problem hiding this comment.
Dropped the number from the doc actually, we probably don't need this in a README. It now only says the image is not a release builder because it'd require a newer glibc.
a739729 to
2c4a3a7
Compare
…both jars on Alpine Motivation: CI builds boringssl-static only on the two release images and never builds the FIPS profile. netty#997 could add the musl check to that profile without the matching link changes, and nothing failed until a downstream build did (netty#1008). Modifications: - docker/Dockerfile.debian13 + docker-compose.debian-13.yaml, services `build` and `build-fips`. Every input is pinned: base image by digest, Debian archive by snapshot.debian.org timestamp, clang by version, Go by version and checksum (BoringSSL's go.mod is ahead of trixie's Go). trixie has no JDK 8, so the build runs on JDK 21 with the pom's --release 8. - ci-pr.yml / ci-build.yml: legs debian13-x86_64 and debian13-x86_64-fips, plus bare-Alpine musl-verify legs on their jars. The FIPS module's integrity check runs in an ELF constructor at dlopen, so only a real load proves the post-link patchelf left it intact. Result: A change to the FIPS profile, or one to the release profiles that is not ported to it, fails on the PR. Not a release image: its artifacts need a newer glibc than the release ones.
2c4a3a7 to
e5ce588
Compare
3 participants
Follow-up to #997 and #1008.
Motivation
#1008 showed the blind spot: nothing in CI builds the FIPS profile, so a change to the release profiles that is not ported to it only fails on someone's downstream build. Building it in CI turned up more: the profile no longer builds on a fresh machine at all (the FIPS tarball bucket is private now) and does not link on aarch64.
Changes, one commit each
buildservice has failed since Make the Linux artifacts loadable on musl (Alpine) again #997.fips-20260721, Google's newest FIPS branch (the tarball atcommondatastorage.googleapis.com/chromium-boringssl-fipsreturns 403 now);CMAKE_POSITION_INDEPENDENT_CODEis back (dropped in Upgrade to BoringSSL 6d503ae1 #950, breaks the aarch64 link); ninja builds onlycrypto ssl decrepit bssl; the compiler is a property (fipsCC/fipsCXX, defaultclang).strtolredirects: it loads on Alpine butlddfails.-Wl,--gc-sectionson the FIPS link line and five weak fallbacks inmusl_compat.c, measured with an empty file. No-op on the CentOS release images.debian13-x86_64anddebian13-x86_64-fips, and bare-Alpinemusl-verifylegs on both jars. The FIPS one matters most: the module's integrity check runs in an ELF constructor atdlopen, so only a real load proves the post-link patchelf left it intact.Verification
All checks pass on this head. Both profiles were also built on amd64 outside CI: zero musl-check warnings, both jars load on bare Alpine, the FIPS one also with gcompat.